Multiple Database Servers

FMOS supports deploying multiple database servers within a single ecosystem. This provides a mechanism for rapid recovery from a failure of the primary database machine by automatically replicating all data to one or more standby servers.

Prerequisites

Before beginning the setup process for additional database servers, the ecosystem must be prepared properly.

Fully Distributed Ecosystem

Multi-database mode is only available for fully distributed ecosystems. Specifically, there must be exactly one machine with the CA, database, and enterprise search roles (the primary database machine), one or more other machines with the database and ES roles (the standby database machines), and at least one separate machine with the AS role. It is not possible to enable multi-database mode in partially distributed or single-server deployments; multi-database mode cannot be enabled if the same machine holds both the DB and AS roles.

Multi-database mode can be enabled when first deploying a new ecosystem, or after an ecosystem has been fully deployed and is operational. In other words, it is possible to add standby database machines to the ecosystem before or after application servers.

Disk Storage Space

When first enabling multi-database mode, FMOS needs to reorganize the files stored on the local filesystem to make them available via the clustered filesystem. The /var volume on the primary database machine needs to have at least 50% free space before beginning.

Network Communication

Database servers do not need to be deployed on the same network or subnet, however, sufficient bandwidth must be available between the primary server and each standby to ensure that changes are replicated as quickly as possible. Replication delay can result in data loss if the primary server fails before all standby servers have received and committed all changes.

Machines holding the AS role need to be able to resolve all DB machines, including standby servers, by Fully Qualified Domain Name (FQDN), and must be able to communicate with them using the following services (IPv4 and IPv6 are both supported):

  • ICMP/ICMPv6
  • IP protocol 50 (ESP)
  • TCP ports 5432 (PostgreSQL) and 55555 (FMOS Control Panel)
  • UDP ports 500 (ISAKMP) and 4500 (IPsec-NAT-T)

The same requirements apply to database machines as well; standby servers must be able to communicate with the primary server using the same services.

Superior / Subordinate Servers

When operating in multi-database mode, the hierarchy of server roles change slightly. The first appliance created in the ecosystem holds the Certificate Authority (CA) role and becomes the superior server to all DB and AS machines. The CA role is responsible for issuing security certificates that are used by the AS, DB, and ES machines to assert their identity to one another and establish encrypted communication channels.

 

 

Step 1: Setup

Provisioning machines for deployment into an FMOS ecosystem with multiple database servers follows the same process as for a standard multi-server ecosystem.

  1. Boot from FMOS installation media (example: DVD-ROM or USB flash drive) and start the automatic installer
  2. Use the Initial Configuration Wizard to set the host name, network connection settings, and user credentials.
  3. On the machine designated as the primary database server, choose New Deployment and Database Only on the ecosystem selection screen.
  4. On all other machines (including standby database servers, application servers, and data collectors), select Existing Deployment.

Step 2: AS Machine: Enter Maintenance Mode

If any machines in the ecosystem already hold the AS role, they must be placed in maintenance mode:

fmos maintenance begin

Step 3: CA Machine: Enable Multi-DB Mode

Once all AS machines (if any) have successfully entered maintenance mode, multiple-database mode can be enabled for the ecosystem by executing the following command on the primary database server:

fmos ecosystem enable-multi-db

The tool will adjust the FMOS System Configuration to enable multi-database mode, and then apply the FMOS Configuration Policy. During this process, FMOS will create a clustered filesystem. Before the clustered filesystem can be used, at least one other database machine must join the ecosystem.

When FMOS is ready for additional database machines to join the ecosystem it will pause and print this message: Waiting for clustered filesystem to be initialized.

Step 4: Standby DB Machine: Join the Ecosystem

At this point, the standby database machines can be added to the ecosystem. As with all machines, this process is handled by the fmos ecosystem join command. When adding a standby database machine to the ecosystem, both the DB and ES roles must be added, and no other role can be added.

fmos ecosystem join firemon-db.yourcompany.com

Joining the ecosystem may take some time, depending on the size of the database and the amount of activity within the ecosystem. An initial synchronization with the master is performed, and then streaming replication is enabled to keep the standby up-to-date. The process will complete once the standby database becomes consistent with the master.

When adding multiple standby servers to the ecosystem, be sure the pg_num_standby_servers variable is set to the correct value on the primary database server. By default, it is set to 1, meaning there can only be one standby server.

Step 5: CA Machine: Finish Deployment

Once at least one standby database machine has joined the ecosystem, the fmos ecosystem enable-multi-db command will resume on the primary database server. It will finish setting up the clustered filesystem, and then move the data from the local filesystem. When this is complete, the AS machines can be brought out of maintenance mode and returned to service.

Step 6: AS Machine: Exit Maintenance Mode

To bring the application servers back online, use the fmos maintenance end command. This will mount the shared filesystem and start the Security Manager services.